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Paul  F.  Levy 
Executive  Director 

Massachusetts  Water  Resources  Authority 
100  First  Avenue 

Charlestown,  Massachusetts  02129 
Dear  Mr.  Levy: 

We  are  writing  to  provide  you  and  members  of  the  Board  with  our 
findings  and  recommendations  regarding  the  procurement  of 
Program  and  Construction  Management  Services  (P/CM)  for  Deer 
Island-Related  Construction. 

We  want  to  thank  you  for  your  cooperation  and  assistance  during 
this  phase  of  the  joint  review  by  our  Offices.     You  have  made 
available  for  this  review  all  documents  that  were  requested, 
and  your  staff  has  cooperated  in  providing  information  over  the 
course  of  many  hours  of  interviews.     We  appreciate  the  courtesy 
and  candor  displayed  by  you  and  your  top  staff. 

At  this  stage  in  our  review,  we  have  found  no  evidence  that 
leads  us  to  conclude  that  any  individual  violated  any  criminal 
statute . 


We  have,  however,  found  serious  flaws  in  the  procurement 
procedures  that  the  Authority  has  utilized  for  this  contract. 
These  procedures  and  the  resulting  contract  are  vulnerable  to 
waste  and  abuse. 


This  letter  summarizes  the  deficiencies  we  have  found  in  the 
Authority's  procurement  of  the  P/CM  contract,  and  concludes 
with  our  recommendations. 


Paul  F.  Levy,  Executive  Director 
Massachusetts  Water  Resources  Authority 
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In  procuring  this  contract,  the  Authority  for  the  most  part 
followed  procedures  originally  adopted  in  1985.  These 
procedures  are,  according  to  Authority  staff,  utilized  for 
virtually  all  procurements  of  services  by  the  Authority. 

The  procedures  call  for  a  two-step  process:     a  Request  for 
Qualifications  (RFQ)  phase,   in  which  the  qualifications  of  all 
competing  firms  are  evaluated,  and  the  number  of  competitors  is 
reduced  (or  "short-listed") ;  and  a  Request  for  Proposals  (RFP) 
phase,   in  which  the  short-listed  firms  are  invited  to  submit 
proposals   (including  proposed  fees)   for  providing  the  services 
sought  by  the  Authority.     We  found  deficiencies  in  both  phases 
of  the  P/CM  procurement;  many  of  these  deficiencies  are 
attributable  to  weaknesses  in  the  Authority's  1985  procedures. 

The  RFQ  Phase 

1 .     The  Request  for  Qualifications  failed  to  set  forth  all 
of  the  evaluation  criteria  utilized  bv  the  Authority. 

The  22-page  RFQ  issued  on  October  30,   1987,  set  forth,   in  half 
a  page,  six  criteria  for  judging  the  qualifications  of 
competing  firms.     The  criteria  were  extremely  general.  For 
example,  one  of  the  six  criteria  consisted,   in  its  entirety,  of 
the  following:   "Qualifications  of  the  project  manager  and  the 
specific  individuals  committed  to  the  project."    The  list  of 
criteria  offered  no  further  explanation  as  to  exactly  what 
qualifications  the  Authority  deemed  necessary. 

In  fact,  the  seven-member  selection  committee  utilized  fifty- 
two  subcriteria  in  rating  the  firms  which  responded  to  the  RFQ. 
For  example,  each  firm's  proposed  project  manager  was  judged, 
among  other  things,  on  the  basis  of  "MIS  [management 
information  systems]  knowledge  and  experience"  as  well  as 
"local  familiarity,"  two  subcriteria  listed  on  the  selection 
committee's  score  sheets  but  not  provided  to  the  competing 
firms.     While  it  is  arguable  that  these  criteria  might  have 
been  inferable  from  the  project  description  contained  elsewhere 
in  the  RFQ,  nothing  in  the  RFQ  put  competitors  clearly  on 
notice  that  the  Authority  wished  to  see  these  attributes 
personified  in  the  project  manager. 

Another  criterion  stated  in  the  RFQ  was  the  following:  "Proven 
track  record  in  meeting  schedules  and  cost  control . "  A 
subcriterion  appearing  in  the  committee's  score  sheets  is, 
"Experience  in  court-ordered  schedules."    While  obviously  the 
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Authority  is  under  a  court  order,  and  the  RFQ  so  stated, 
nothing  in  the  RFQ  suggested  that  a  firm's  on-time  record  would 
be  downgraded  if  achieved  in  the  absence  of  a  judicial  decree. 
Another  criterion  stated  in  the  RFQ  was  as  follows: 
"Experience  of  the  firm  (and/or  Joint  Venture  team  members)  in 
the  management  of  similar  scale  construction  projects 
particularly  those  involving  .   .    .  underwater  tunnel 
construction  ..."    The  selection  committee  utilized  three 
subcriteria  involving  experience  of  the  firm,  the  project 
manager,  and  key  staff  with  "hard  rock  tunnels."  Underwater 
tunnels  may  not  involve  hard  rock.     If  the  Authority  wished  to 
limit  its  view  to  hard  rock  tunnels,  or  to  downgrade  a  firm  for 
failing  to  provide  information  on  hard  rock  tunnel  construction 
experience,  the  Authority's  criteria  should  have  said  so.  They 
did  not. 

A  procurement  system  is  not  fair  if  it  fails  to  enunciate  in 
advance,  clearly  and  publicly,  the  criteria  by  which 
competitors  are  to  be  judged.     At  best,  competitors  are  left  to 
guess  at  the  rules,  and  well-qualified  competitors  may  be 
downgraded  simply  for  guessing  wrong;  at  worst,  the  system  puts 
a  premium  on  inside  information. 

2 .  Some  of  the  RFQ  criteria  and  subcriteria  were  too  vague 
to  be  applied  consistently  and  fairly. 

Some  of  the  written  criteria  and  subcriteria  were  too  vague  to 
provide  useful  standards  for  comparing  the  qualifications  of 
competing  firms.     The  selection  committee  introduced 
distinctions  that  were  not  articulated  in  either  the  criteria 
or  subcriteria. 

In  particular,   firms  wishing  to  respond  to  the  RFQ  were 
required  to  submit  a  five-page  essay  describing  —  in  advance 
of  seeing  the  Authority's  RFP  —  the  firms'  "management 
approaches  intended  to  maintain  the  court-ordered  schedule, 
control  costs,  mitigate  claims,  maintain  community  relations, 
and  resolve  logistical  constraints  ..."     The  firms  were 
further  required  to  submit  one-page  statements  "commenting  on 
or  suggesting  approaches  to"  eight  "issues"  which  the  Authority 
itself  had  apparently  not  resolved.     One  of  these  issues  was 
"Incentive-based  compensation  for  the  P/CM"   (that  is,  how  the 
firm  felt  about  one  approach  to  getting  paid) ;  another  was  "The 
program  to  maintain  positive  community  relations"  —  a  program 
yet  to  be  defined  by  the  Authority. 
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It  is  difficult  to  see  how  the  Authority  could  have  fairly 
utilized  these  responses.     The  applicable  selection  criterion, 
as  stated  in  the  RFQ,  was  hopelessly  vague:  "Specific 
understanding  of  the  program  and  construction  management  task 
and  knowledge  of  the  area  as  demonstrated  in  the  quality  of  the 
proposed  approach."     The  applicable  subcriteria  were  equally 
standardless:     "Responses  to  special  issues  [followed  by  a  list 
of  the  issues,  e.g.,   "positive  community  relations"]"; 
"Familiarity  with  regulatory  agencies";     "Familiarity  with 
local  construction" . 

It  would  have  been  appropriate  for  the  Authority  to  seek  and 
evaluate,  on  the  basis  of  stated  criteria,  construction 
experience  within  Massachusetts.     It  was  not  appropriate, 
however,   for  the  Authority  to  compare  "responses"  to  vaguely 
worded  "issues,"  nor  to  evaluate  and  compare  actual  experience 
(or,  as  phrased  in  the  subcriteria,   "familiarity")   from  those 
brief  responses. 

The  firms  had  not  been  provided  with  a  detailed  RFP,  setting 
out  precisely  the  services  sought  by  the  Authority  and  the 
standards  by  which  the  Authority  would  evaluate  proposed 
approaches  to  providing  exactly  those  services;  without  clearer 
guidance  from  the  Authority,  these  sketchy  submissions  amounted 
to  a  standardless  essay  contest.     Firms  were  apparently 
downgraded  if  their  responses  were  deemed  by  the  committee  to 
be  too  "generic." 

3 .     The  Authority  failed  to  evaluate  the  financial 
stability  of  firms  in  the  RFQ  phase. 

The  Authority's  1985  rules  call  for  "financial  stability"  of 
competing  firms  to  be  evaluated  in  the  RFQ  phase.     In  the  P/CM 
procurement,  the  evaluation  of  financial  stability  was  not 
undertaken  in  the  RFQ  phase.     As  explained  by  the  chairman  of 
the  selection  committee,  the  Authority  omitted  from  the  RFQ,  by 
mistake,  a  requirement  that  firms  submit  audited  financial 
statements  upon  which  an  evaluation  could  have  been  based. 
Therefore,  consideration  of  financial  stability  was  deferred 
until  the  RFP  stage. 

The  omission  —  which  would  have  been  easily  corrected  in  a 
less  vulnerable  selection  process  —  may  have  contributed  to 
the  unfairness  of  this  procurement.     The  omission  undoubtedly 
had  the  effect  of  advantaging  the  firm  which  was  ultimately 
selected.     But  whether  that  effect  would  have  been  large  or 
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small  is  a  matter  of  pure  speculation.     Had  all  of  the  firms 
been  rated  and  compared  on  the  basis  of  financial  information, 
the  committee's  overall  judgment  as  to  the  ranking,  and  indeed 
the  decision  as  to  how  many  firms  to  allow  into  the  RFP  phase, 
might  have  been  different.     Certainly,   financial  stability 
ought  to  be  a  consideration  in  determining  whether  a  firm  is 
qualified  for  a  large  public  contract. 

Whatever  the  effect  of  the  omission,  the  criterion  itself 
further  illustrates  the  lack  of  clear  standards  in  the 
Authority's  procedures.     Although  the  Authority's  rules  call 
for  "financial  stability"  to  be  evaluated,  they  provide  no  clue 
as  to  what  level  of  financial  stability  is  required.  Indeed, 
Authority  staff  members  acknowledged  in  an  interview  that,   if  a 
firm  had  not  declared  bankruptcy,  they  were  uncertain  as  to 
whether  the  Authority  should  or  could  exclude  the  firm  from 
competing  for  a  contract.     Following  the  Authority's  rules 
would  simply  have  added  one  more  standardless  criterion  to  the 
mix. 

4 .     The  Authority's  method  of  rating  and  ranking 
qualifications  was  vulnerable  to  arbitrary  decisions. 

The  selection  committee  relied  heavily  on  a  numerical  rating 
system.     Such  systems  have  a  tendency  to  focus  the  attention  of 
evaluators  and  the  ultimate  decision-makers  on  numbers  rather 
than  on  the  appropriateness  of  the  judgments  underlying  the 
numerical  scores.     As  used  in  this  procurement,  the 
overreliance  on  numerical  scores  proved  particularly  vulnerable 
for  several  reasons. 

First,  the  evaluation  committee  members  were  provided  with  no 
written  instructions  as  to  how  to  score  each  criterion.  For 
example,  there  was  no  indication  of  what  it  might  take  to  get  a 
perfect  score  on  any  criterion,  nor  were  evaluators  told 
whether  it  would  be  appropriate  or  inappropriate  to  grade  on  a 
curve  (that  is,  giving  a  high  score  to  the  best  of  a  bad  lot  on 
any  criterion) .     Coupled  with  vaguely  worded  criteria,  the 
absence  of  clear  guidance  left  each  rater  to  devise  major 
elements  of  the  grading  procedure.     One  member  assigned  points 
to  each  subcriterion;  the  others  awarded  points  only  for  each 
general  criterion.     This  lack  of  guidance  was  particularly 
troublesome  because  of  the  make-up  of  the  selection  committee. 
Four  members  had  engineering  backgrounds;  three  did  not.  Yet 
the  three  members  having  no  engineering  background  were  asked, 
without  written  instructions,  to  evaluate  the  quality  of  a 
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firm's  experience  and  award  numerical  points  on  such  criteria 
as  "large  diameter,  hard  rock  tunnels,"  "marine  construction," 
and  "value  engineering"  experience.     A  rater's  assignment  of 
points  under  these  conditions  is  almost  inevitably  arbitrary. 

Second,   several  committee  members'  evaluations  consisted  of 
little  (or  no)  more  than  the  bare  numbers.     The  committee 
chairman  produced  twenty-one  pages  of  typed,  detailed  notes, 
and  two  other  members  made  extensive  notations  on  their  score 
sheets.     But  the  majority  left  a  scantier  record.     Two  members 
made  a  few  notes  and  check  marks  on  their  score  sheets;  two 
other  members  contributed  no  notes  whatever.     Moreover,  one  of 
the  members  who  made  careful  notes  in  his  original  scoring 
subsequently  changed  his  mind,   in  the  course  of  committee 
discussions,  but  left  no  record  as  to  why.     A  selection  system 
which  focuses  on  numerical  point  awards,  while  tolerating  the 
absence  of  clear,  written  explanations  for  those  ratings  by  the 
responsible  public  officials,   invites  arbitrary  and 
unaccountable  decisions. 

Third,  the  scoring  methodology  introduced  a  conversion  factor 
which  increased  the  likelihood  of  an  arbitrary  decision.  The 
committee  converted  each  member's  scores  for  each  of  the  six 
competing  firms  (these  raw  scores  were  on  a  100-point  scale, 
with  higher  scores  for  "better"  qualifications)  to  a  numerical 
rank  order.     That  is,  whatever  the  raw  score  assigned  by  a 
member  to  the  firm  he  rated  most  highly,  that  score  was  simply 
converted  to  a  new  six-point  scale;  that  member's  top-rated 
firm  was  assigned  1  point,  his  second-ranked  firm  was  assigned 
2  points,  etc.     Every  firm  thus  received  from  one  to  six 
points.     On  the  new  scale,  the  "better"  the  firm,  the  lower  its 
points.     The  committee  then  added  the  points  each  member 
awarded  to  each  firm,  using  the  six-point  scale,  to  arrive  at 
total  points.     The  three  firms  with  the  lowest  totals  won  the 
right  to  participate  in  the  RFP  phase. 

The  problem  with  such  a  mechanical  transposition  is  that  it  is 
utterly  arbitrary.     The  arbitrariness  is  particularly  acute  if 
the  purpose  of  the  scoring  system  —  as  in  this  instance  —  is 
to  narrow  a  pool  of  competitors  down  to  a  predetermined  number. 
If  the  raw  scores  are  far  apart,  the  transposition  will  make 
the  competitors  appear  more  closely  matched;     if,  as  was  the 
case  here,  the  raw  scores  are  relatively  close,  the 
transposition  may  exaggerate  the  differences  among  competitors. 
The  transposition  may  also  result  in  a  change  in  ranking.  In 
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this  case,  although  the  documentation  is  too  sparse  to  be 
certain,   it  appears  that  arbitrarily  converting  the  points  from 
one  scale  to  another  may  have  determined  which  firm  came  in 
third  —  and  thus  was  able  to  submit  a  proposal .     The  chart 
which  follows  shows  the  total  raw  scores  of  committee  members, 
from  their  score  sheets,  prior  to  discussions  during  which  one 
committee  member  apparently  changed  his  mind  about  some  of  his 
scores : 


Total  Raw  Scores 
Boston 

Harbor  Boston 
Program  Morrison  Harbor    Rust  Sverdrup 

Criteria  Managers  Kaiser  Knudson  Partners  Intl .   &  Parcel 


Experience  (Max=150) 

125 

126 

110 

113 

114 

115 

Project  Manager 

Qualifications 

(Max=105) 

84 

83 

77 

85 

75 

68 

Staff  Qualifications 

(Max=70) 

55 

56 

53 

60 

56 

46 

Scheduling  &  Cost 

Controls  (Max=140) 

113 

112 

102 

103 

106 

103 

Understanding 

(Max=105) 

79 

93 

87 

90 

83 

77 

Ability  to  Work 

Together  (Max=35) 

29 

31 

28 

21 

28 

27 

Affirmative  Action 

Program  (Max=70) 

40 

54 

43 

61 

48 

44 

Total  Score  (Max=675) 

525 

555 

500 

533 

510 

480 

Percent  of  Maximum 

Score 

77.8% 

82.2% 

74. 1% 

79.0% 

75.  6% 

71.1% 

Because  of  inadequate  documentation,   it  is  not  possible  to 
establish  how  these  raw  scores  would  have  changed  when  one 
committee  member  changed  his  mind;     new  scores  simply  were  not 
recorded.     However,  this  snapshot  of  the  scoring  process 
suggests  a  relatively  close  race  (particularly  given  the 
absence  of  clear  standards  and  instructions) . 
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Under  the  conversion  formula,  the  top  three  firms  (and  the 
converted  points  awarded)  were  as  follows:     Kaiser  Engineers, 
Inc.    (13) ;  Harbor  Partners  (20) ;  and  Rust  International   (25) . 
The  fourth-  and  fifth-place  finishers  came  within  only  3  points 
of  Rust  International  (the  change  in  rankings  made  by  one 
committee  member  did  not  affect  the  first-  and  second-place 
choices  on  either  scale) .     Kaiser  appears  as  a  result  of  the 
conversion  to  have  markedly  superior  qualifications.     Thus,  it 
appears  that  negligible  differences  among  firms  were 
exaggerated  by  use  of  the  conversion  formula. 

Our  conclusion  is  that  the  raw  scores  were  themselves 
unreliable.     Applying  the  conversion  formula  amounted  to 
holding  these  scores  up  to  a  circus  mirror  and  basing  the  final 
decision  on  the  reflected  image.     A  bad  situation  was  made 
worse. 


Such  mechanical  rating  manipulations,  however  well- 
intentioned,  have  no  place  in  a  prudent,  soundly  reasoned, 
we 11 -documented  procurement  system. 

5 .  The  selection  process  was  inadequately  documented. 

The  failure  of  individual  selection  committee  members  to  record 
the  bases  for  their  ratings  undermines  the  reliability  of  those 
ratings.     Indeed,   in  the  absence  of  any  other  record,  the 
numbers  (which  represent  attempts  to  quantify  various  judgments 
which  may  not  be  susceptible  to  quantification)  become  all- 
important.     In  a  prudent  and  accountable  procurement  process, 
what  matters  is  the  care  and  persuasiveness  of  the  reasoning 
underlying  the  selection  of  a  vendor.     In  this  procurement,  the 
record  of  that  reasoning  is  characterized  by  large  gaps. 

The  minutes  of  the  two  meetings  of  the  selection  committee  are 
brief  and  inadequate.     A  two-hour  discussion  that  produced  the 
ultimate  ranking  of  the  six  firms  on  the  basis  of  fifty-two 
subcriteria  is  reduced  in  the  minutes  to  fourteen  brief 
comments . 


An  accountable  process  requires  clear  and  detailed 
documentation  at  every  step. 
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6 .     The  elimination  of  three  firms  at  the  RFO  stage  unduly 
limited  competition  for  the  P/CM  contract. 

A  sound  public  procurement  process  seeks  to  maximize 
competition  among  qualified  firms.     Unwarranted  restrictions  on 
competition  risk  inflated  costs,  poorer  services,  and 
vulnerability  to  waste  and  abuse. 

The  RFQ  process  utilized  by  the  Authority  did  not  yield  a 
convincing  basis  for  eliminating  any  of  the  competing  firms. 
Indeed,  the  chairman  of  the  selection  committee  stated  in 
interviews  that  he  regarded  all  six  firms  to  be  "qualified." 
The  Authority  failed  to  enunciate  clear,  objective  standards 
for  distinguishing  between  qualified  and  unqualified  firms. 
Such  standards  should  have  been  articulated  in  this 
procurement,  and  would  have  adequately  protected  the  Authority 
from  doing  business  with  any  unqualified  firm. 

Instead,  the  Authority  listed  vague  criteria  by  which  it  hoped 
to  compare  the  relative  merits  of  competitors,  in  order  to 
achieve  the  arbitrary  goal  of  reducing  the  competition  to  three 
or  four  firms. 

The  most  basic  flaw  in  the  RFQ  phase  was  that  this  goal  —  to 
reduce  competition  to  a  predetermined  range  —  was  contrary  to 
the  best  interests  of  the  Authority  and  its  rate-payers. 

In  our  view,  the  benefit  to  the  Authority  of  restricting  the 
competition  was  a  negligible  time  saving  in  the  Authority's 
review  of  proposals.     The  cost  would  appear  far  greater:  the 
lost  opportunity  to  consider  proposals  (and  proposed  fees)  from 
other  qualified  firms. 

The  RFP  Phase 

1.     The  Request  for  Proposals  failed  to  set  forth  all  of 
the  evaluation  criteria  utilized  by  the  Authority. 

The  RFP  contained  the  same  general  evaluation  criteria  as 
appeared  in  the  RFQ,  with  two  additions:     the  cost  proposed  by 
each  firm,  and  "familiarity  in  dealing  with  the  requirements  of 
regulatory  agencies"   (which  had  been  a  subcriterion  in  the  RFQ 
stage) . 

For  the  most  part,  the  subcriteria  remained  the  same,  as  well, 
although  the  list  of  subcriteria  grew  to  sixty-three.     Just  as 
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the  RFQ  subcriteria  were  unavailable  to  respondents,  the  RFP 
subcriteria  were  not  disclosed  to  proposers  in  the  RFP. 

2 .     Some  of  the  RFP  criteria  and  subcriteria  were  too 
vague  to  be  applied  consistently  and  fairlv. 

The  criteria  and  subcriteria  from  the  RFQ  phase  remained  just 
as  vague  when  the  process  moved  to  the  RFP  stage.     For  example, 
one  criterion  in  the  RFP  was  as  follows:  "Specific 
understanding  of  the  construction  management  task  and  knowledge 
of  the  area  as  demonstrated  in  the  quality  of  the  proposed 
approach  and  scope  of  the  work."    Vague  as  this  criterion  is, 
it  was  further  confounded  by  the  addition  of  a  new 
subcriterion:     "quality  of  additions  to  the  scope"  of  work  in 
the  RFP  [emphasis  added] . 

One  of  the  bedrock  principles  of  prudent  public  procurement  is 
that  the  awarding  authority  should  define  its  own  needs  and 
then  seek  a  vendor  to  fulfill  those  needs;  it  is  not  conducive 
to  fair  competition  or  prudent  cost  control  to  invite 
prospective  vendors  to  write  their  own  specifications  for  some 
or  all  of  the  work. 

This  procurement  departed  from  that  principle.     The  RFP  stated: 
"The  proposer  shall  feel  free  to  propose  additional  activities 
that  are  felt  [by  the  proposer]  to  be  necessary  to  provide 
effective  management  of  the  planning  and  design  of  the  Deer 
Island  projects." 

This  invitation  led  to  the  standardless  attempt  by  the 
Authority  to  compare  the  "quality"  of  add-on  services  that  were 
not  identified  in  the  RFP. 

Perhaps  symptomatic  of  the  ill-defined  evaluation  criteria  was 
the  ambiguity  in  the  criterion  of  Minority  and  Women  Business 
Enterprise  Participation  (MBE/WBE) .     Although  the  Authority's 
affirmative  action  policies  and  goals  were  set  forth  in 
considerable  detail,  the  precise  evaluation  criteria  remained 
vague.     The  RFP  said  that  firms  would  be  evaluated  for 
"Creativity  in  complying  with  the  Authority's  Affirmative 
Action  Program";  no  definition  of  "creativity"  was  offered. 
The  subcriteria  utilized  by  the  selection  committee  omitted  any 
reference  to  creativity,  and  instead  referred  to  the  "extent" 
to  which  minimum  goals  were  exceeded;  "commitment  of  minorities 
and  women  to  senior  level  positions";  and  "work  profiles  of  the 
proposed  teams."    No  further  written  instructions  guided  the 
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evaluators.     In  the  end,  the  critical  inquiry  became  one  of 
whether  the  winning  firm  had  met  the  bare  minimum  requirement  - 
-  and  that  inquiry  suggested  that  the  minimum  requirement  was 
ambiguous.     A  review  of  the  proposals  by  Touche  Ross  &  Co., 
commissioned  by  the  Authority  as  part  of  the  RFP  process, 
indicates  three  possible  ways  of  calculating  compliance  with 
the  Authority's  MBE/WBE  participation  requirements;  Kaiser's 
proposal  was  determined  by  Touche  Ross  to  be  deficient  under 
two  of  the  methodologies.     The  Authority  apparently  elected  to 
utilize  the  third. 

3 .  The  RFP  failed  to  define  clearly  the  services  sought  by 
the  Authority. 

In  significant  areas,  the  RFP  contained  unacceptably  vague 
descriptions  of  the  services  to  be  provided.     For  example, 
proposers  were  required  to  describe  how  they  would  "assist  the 
Authority  in  determining  and  executing  the  most  effective 
manner  of  providing  key  construction  support  services  such  as 
the  operation  of  a  water  transportation  system  ..."  Firms 
were  referred  to  Appendix  G  for  additional  information  on  this 
system.     There  was  no  Appendix  G.     The  Authority  did  not  yet 
know  the  dimensions  of  its  water  transportation  system,  which 
was  to  be  put  out  to  bid  separately;     yet  it  sought  from 
proposers  information  about  how  the  proposer  planned  to 
"assist"  in  operating  an  undefined  system,  and  how  much  the 
proposer  would  charge  for  this  assistance. 

Other  services  were  described  with  similar  vagueness.     The  RFP 
called  for  firms  to  propose  "an  aggressive  community  relations 
program."     The  aims  of  this  program  were  set  out  in  eight 
typewritten  lines.     Firms  were  left  to  define  the  dimensions 
and  cost  of  the  program.     One  proposal  offered  a  $1  million 
program;  the  winning  proposal  offered  a  $2.4  million  program. 

Another  example  of  vagueness  is  the  description  of  the  P/CM's 
responsibilities  regarding  Management  Information  Systems 
(MIS).     The  RFP  calls  for  the  P/CM  to  "develop,  update  and 
maintain"  a  comprehensive  computer  system  that  will  be 
compatible  with  the  internal  system  the  Authority  is 
developing.     The  Authority  is  pledging  to  reimburse  the  P/CM 
for  the  full  cost  of  the  P/CM's  system,   including  hardware  and 
software.     This  is  not  a  prudent  way  of  acquiring  any  MIS 
capability.     The  Authority  does  not  know  what  system  it  will 
acquire  for  itself;  yet  it  now  plans  to  reimburse  the  P/CM  for 


Paul  F.  Levy,  Executive  Director 
Massachusetts  Water  Resources  Authority 
88-4009-3 


March  29,  1988 
Page  12 


an  undefined  system  of  unknown  size  and  complexity  without 
further  competition. 

A  vague  description  of  services  in  an  RFP  undermines  fair 
competition.     It  is  impossible  to  compare  firms  on  an  equal 
footing  if  each  firm  can  write  its  own  specifications  for  the 
services  to  be  performed.     A  vague  RFP  also  invites  waste; 
each  proposer  is  encouraged  to  expand  the  services  (and  the 
cost)  beyond  what  the  Authority  has  ascertained  to  be  its 
needs.     Finally,  a  vague  RFP  opens  the  door  to  favoritism: 
when  neither  the  evaluation  criteria  nor  the  services 
themselves  are  clearly  defined,  awards  based  on  favoritism 
become  harder  to  detect. 

4 .  The  proposed  contract,  as  described  in  the  RFP.  is 
excessively  vulnerable  to  uncontrolled  costs. 

In  at  least  a  half  dozen  respects,  the  proposed  contract  is 
likely  to  prove  vulnerable  to  escalating  costs. 

(a)  Incentive  payments.     Authority  staff  estimated  in  an 
interview  that  up  to  $700,000  under  the  contract  is  subject  to 
payment  on  an  incentive  basis.     That  is,  the  actual  amount  paid 
to  the  firm  as  the  work  progresses  will  be  determined  on  the 
basis  of  various  incentive  factors.     The  factors,  as  suggested 
in  the  RFP,  are  as  ill-defined  as  the  evaluation  criteria. 
They  include  such  nebulous  factors  —  to  be  further  negotiated 
before  the  contract  is  signed  —  as  "speedy  mobilization," 
"effectiveness  in  community  relations,"  "resolution  of 
unforseen  issues,"  and  "effective  management  of  lead  design 
engineer."    What  we  have  seen  of  the  RFQ  and  RFP  process  gives 
cause  for  concern  that  incentive  payments  to  the  P/CM  will  be 
predicated  on  standardless  reviews  of  performance.  The 
Authority  will,  on  each  bill,  choose  between  reductions  from 
the  maximum  payment  allowed  (and  a  confrontation  with  the  P/CM) 
or  full  payment  (and  a  contented  P/CM) .     The  incentive  system 
as  described  in  the  RFP  does  not  appear  to  be  one  that  can  be 
prudently  managed. 

(b)  Allowances.     The  RFP  included  "allowances"  for  many 
services  that  were  not  adequately  defined.     These  allowances 
included  $300,000  for  "project  mobilization,"  $400,000  for  the 
MIS,  $400,000  for  seeking  regulatory  approvals,  $100,000  for 
environmental  analyses  that  may  be  needed,  $75,000  for 
unspecified  "staff  support"  to  the  Authority,  and  $150,000  for 
yet-to-be-defined  "special  activities"  which  may  arise.  The 
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total  of  these  allowances  is  $1.5  million.     All  but  one  of 
these  allowances  reflect  the  Authority's  ballpark  budget 
figures  for  items  it  chose  to  include  in  the  RFP  even  though 
the  Authority  could  not  precisely  describe  the  services  it 
wanted;  the  amounts  the  Authority  will  actually  pay,  and  the 
services  the  Authority  will  actually  receive,  will  be 
determined  through  negotiation.     In  effect,  each  of  these 
services  will  now  be  defined  and  paid  for  on  a  sole-source 
basis.     These  services  should  simply  have  been  excluded  from 
the  RFP. 

The  remaining  item  is  "project  mobilization"  —  the  P/CM's 
costs  for  establishing  an  office  near  the  Authority's.  The 
Authority  promised  in  the  RFP  to  reimburse  this  cost  —  based, 
at  least  in  part,  on  the  P/CM's  internal  policies  for 
relocating  its  employees.     A  more  tightly  written  RFP,  with 
more  clearly  defined  services,  would  not  have  made  this  item 
subject  to  open-ended  negotiation  at  the  end  of  the  selection 
process.     Each  firm  should  have  been  required  to  include  the 
full  costs  of  hiring  that  firm  —  including  mobilization 
costs  —  in  its  proposal. 

(c)  Reimbursables .     Some  of  the  allowance  items  may  be 
negotiated  to  a  fixed  cost;  some  (such  as  project  mobilization 
and  MIS)  were  advertised  in  the  RFP  as  reimbursables. 
Reimbursable  items  are  allowed  elsewhere  in  the  RFP,  as  well: 
computer  charges,  printing,  travel,  insurance,  rent,  and 
utilities.     Reimbursable  items  invite  higher  costs.     There  is 
no  incentive  for  the  vendor  to  refrain  from  incurring  the  cost, 
or  to  seek  the  lowest  cost,  if  the  expense  is  to  be  passed 
along  to  the  Authority.     Moreover,  verifying  the  accuracy  and 
legitimacy  of  a  reimbursable  expense  puts  on  the  Authority  the 
burden  of  auditing  the  expense.     The  total  amount  of 
reimbursables  is,  of  course,  unknown  (and  perhaps,  under  the 
terms  of  this  RFP,  presently  unknowable) .     The  estimates  for 
just  two  items  —  project  mobilization  and  MIS  —  total 
$700,000.     The  eventual  amount  is  likely  to  be  significantly 
higher. 

(d)  Cost  plus  fixed  fee.     The  basic  fee  structure  of  the 
entire  contract  invites  overruns.     The  P/CM  is  to  be  paid  on  a 
cost-plus-fixed-fee  basis.     The  "cost"  component  of  the 
Authority's  payments  is  based  on  labor  and  overhead  rates 
proposed  by  the  P/CM.     Each  firm  also  submitted  an  estimated 
"level  of  effort"  for  each  task  in  the  RFP  —  that  is,  the 
firm's  own  estimate  of  the  number  of  person-hours  various 
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employees  of  the  firm  will  take  to  perform  each  task.  The 
problem  is  that  many  of  the  tasks  in  this  contract  are  too  ill- 
defined  to  permit  much  confidence  in  the  accuracy  of  the 
estimated  levels  of  effort;  had  the  tasks  been  more  completely 
defined,  the  firms  could  have  been  required  to  submit  prices 
for  completing  each  task.     Since  the  Authority  has  not 
described  precisely  what  it  wants  (for  example,  what 
constitutes  an  "aggressive  community  relations  program,"  or 
what  assistance  it  needs  with  the  transportation  system) ,  the 
Authority  has  little  basis  for  insisting  that  the  P/CM  complete 
the  necessary  tasks  in  strict  conformity  with  the  estimated 
levels  of  effort.     There  is  a  high  probability  that  costs  under 
this  contract  will  significantly  exceed  present  estimates. 

(e)  Negotiations .     Given  the  vague  scope  of  services,  the 
unreliability  of  cost  estimates,  the  undefined  criteria  for 
incentive  fees,  and  the  imprecision  about  exactly  what  expenses 
are  reimbursable,  there  is  little  of  the  P/CM  contract  that  is 
not  subject  to  negotiations.     Such  open-ended  negotiations  are 
a  no-win  situation  for  the  Authority.     If  the  Authority 
negotiates  a  significant  reduction  in  the  scope  of  the  contract 
by  eliminating  some  of  the  least  cost-controlled  aspects,  the 
result  would  be  a  different  contract  from  the  one  that  was 
advertised;  such  a  change,  in  itself,  would  constitute  a  mark 
of  an  unfair  procurement  process,  since  competing  firms  might 
have  approached  the  process  differently  had  the  new  terms  been 
advertised.     On  the  other  hand,  if  the  Authority  presses  ahead 
with  the  contract  as  outlined  in  the  RFP,  cost  overruns  are 
likely.     In  a  tightly  structured  procurement  process,  the 
boundaries  of  negotiations  should  be  far  more  limited. 

(f)  Extension.     The  proposed  contract  will  last  for  three 
years.     Authority  staff,  in  interviews,  estimated  that  the 
selected  firm  would  have  approximately  50  persons  on  the 
contract  payroll.     But  that  is  only  the  beginning.  P/CM 
services  will  be  needed  for  several  years  after  the  current 
contract;  indeed,  once  construction  starts  —  four  years  from 
now  —  Authority  staff  estimated  that  the  P/CM  would  add 
another  200  persons  to  the  payroll.     Unless  the  Authority 
begins  planning  now,  two  developments  are  virtually  certain: 
the  P/CM  will  be  awarded  a  new  or  extended  contract  at  the 
expiration  of  this  contract;  and  the  P/CM's  fees  will  soar. 
Those  fees  will  far  exceed  the  amounts  the  Authority  would 
incur  by  adding,  three  years  from  now,  equivalent  positions  to 
its  own  payroll  (to  the  extent  prudently  necessary)   instead  of 
paying  the  P/CM  to  increase  its  payroll.     To  avoid  enormous 


Paul  F.  Levy,  Executive  Director 
Massachusetts  Water  Resources  Authority 
88-4009-3 


March  29,  1988 
Page  15 


fees  later,  the  Authority  should  begin  now  to  plan  a  staffing 
pattern  that  minimizes  the  costs  of  huge  consultant  contracts 
in  the  coming  years.     The  RFP,  while  referring  vaguely  to  a 
plan  for  "transferring  the  final  responsibilities  to  the 
owner's  staff,"  did  not  include  clear  provisions  for  minimizing 
the  expansion  and  cost  of  the  P/CM  arrangement  in  the  years 
following  this  three-year  contract. 


Two  recommendations  follow  from  our  review. 

First,  we  recommend  that  the  Authority  cancel  this  procurement 
and  restart  it  under  new  procedures.     The  Authority's  current 
procurement  procedures  are  fundamentally  deficient.     They  do 
not  adequately  protect  the  Authority  or  its  rate-payers  from 
fraud,  waste,  and  abuse  in  the  award  of  contracts. 

Second,  we  recommend  that  the  Authority  adopt  as  its  procedures 
the  proposed  Uniform  Procurement  Code  which  we  made  available 
to  you  and  is  now  pending  in  the  Legislature.     While  as  drafted 
the  legislation  does  not  apply  to  the  Authority,  we  recommend 
that  the  Authority  adopt  these  procedures. 

We  stand  ready  to  assist  you  and  the  Authority  as  you  proceed. 


Recommendations 


Sincerely, 


cc:  Board  of  Directors 


